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; Twenty instructors from eight junior college districts, meeting at a workshop 
| sponsored by the League for Innovation in the Community College, developed this 
I research paradigm fc r- teachers of .English composition in 2-year colleges. A 
systematic method of inquiry is described first, including (1) the rationale for 
\ deliberate inquiry, (2) methods of selection for instructors and classes to participate 
[ in the study, (3) the development of a scoring key for composition, (4) checking the 
| reliability of raters who will use the key. (5) choosing composition topics, and (6) 

; instructions to give participating students. The implementation of the study is outlined 
| next, including procedures for (1) coordinating the study, (2) administering the 
[ composition test, (3) coding and sampling, (4) distributing the compositions for 
[ scoring. (5) scoring the compositions, and (6) collating the results. Analysis of the 
l data is then discussed, and includes several questions that such an analysis could 
| answer. The final discussion reviews the strengths and weaknesses of the desian* and 
! some of the implications of this type of inquiry for curriculum and instruction. (MG) 
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INTRODUCTION 



The research paradigm outlined in this Topical Paper was designed 
(and is for use) by teachers of English Composition in two-year colleges. 
TVenty instructors from eight junior college districts met at a work- 
shop sponsored by the League for Innovation in the Community College in 
July, 1968, and developed these guidelines. A study using the design 
is being coordinated by the Clearinghouse. Results will be reported in 
a later paper in this series or in a Clearinghouse monograph. 

We hope that a large number of teachers of English wllll be so 
concerned with their students' learning to write that they will set up 
projects along the lines projected here. Groups wishing to employ the 
design in their own studies may receive further Information from the 
Clearinghouse. 



Arthur M. Cohen 

Principal Investigator and Director 
ERIC Clearinghouse for Junior 
College Information 



IS ANYONE LEARNING TO WRITE? 



Introduction 

Does anyone learn to write In college?- - How would one go about seeking 
pertinent data? Ask the instructors? Search the dean's files? Poll the students 
or check their grade point averages? 

All available answers suffer from limitations occasioned by bias, distorted 
perception, and, above all, inadequate available information. Grade marks 
earned in freshman composition courses, for example, say little about writing 
ability. Classes may be based on traditional grammar, structural linguistics, 
literature, rhetoric, logic, semantics, communication in the mass media, public 
speaking, or on any combination thereof- (5). In some cases instruction and prac- 
tice in writing are not included in the course at all and marks assigned are 
related instead to verbal performance, responses to quick score exams, or facil- 
ity in classroom discussion. 

Other sources shed no light whatsoever on the question of student learning. 
Deans' files typically include data on numbers of students who transfer to four 
year institutions and the grades they earn. When students and instructors are 
queried, their answers are usually, "I feel I learned to write” and "I think our 
students are writing better now than they were when they enrolled.” The extent 
to which the questioner is satisfied with those types of responses depends upon 
his faith in the accuracy of student memory and instructor perception. 

In short, despite the notation in every college catalog, "The student will 
learn to write effectively," no one really knows the extent to which student 
writing improves, or if, in fact, it improves at all as a result of college atten- 
dance. He who itould seek an answer must design and conduct his own investigation. 
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Th« print requisite for arueh a study Is « deliberate method of inquiry that nust 

ba built on a composition scoring device. 

latlonala 

this design atoms from certain definitions and philosophical positions 
regarding education. The overriding position is that education is a process of 
aovlng people from one set of capabilities or tendencies to another. Within this 
framework are certain definitions; e.g., learning is changed capability for, or 
tendency toward, acting in particular ways; and instruction is the deliberate 
sequencing of events so that learning occurs. Thus, education is not seen as 
the providing of an environment in which something of unknown effect may or may 
not occur; rather, education is the bringing about of change. Within the comnunlty 
college, it is the Instructors who should predict and define' the nature of that 
change and assess the effects of their instructional processes. 

I. Selection the Participants 

All Instructors of English composition are eligible. Several members of 
one department may be involved; the more who participate, the better the study. 

design works equally well when instructors from more than one college take 
part. 

III. Selecting the Classes 

Each participating Instructor may involve as many students as he chooses. 

8e may bring one or any number of his classes into the project. 

III. Developing A Scoring gey 

Although composition seorlng keys are readily availeble, none should be 
selected unless all participants understand and are willing to use all categories 
{an unlikely eventuality). Agreement is more readily achieved if the Instructors 
meet together and develop their own key. In developing the key, some instructors 
may be forced to give up favored viewpoints in order to achieve a group consensus 
^*7 that is used should be biilt on separate scoring categories 
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rather then on single global measures. There may be as many categories 
as is deemed necessary. 

For scoring ease it is desirable that categories be dichotomous, 
for example: 

1. There are serious or distracting errors in punctuation. Yes No 

2. Each paragraph is developed with supporting details. Yes No 

It should be noted that for some categories in the scoring key, the 
"correct*' answer is "No", as in example one above, and for some it may 
be, "ths"? •» in example two. 

IV. Checking Rater Reliability 

In the course of developing the key, scoring reliability is checked 
by having each participant read and mark duplicate compositions. A cate- 
gory consistently failing to receive 80 percent agreement should be re- 
fined or discarded. That is, if four of five participants cannot agree 
on the meaning of terms as applied to sample compositions ("insightful," 

"well developed," "excessive errors in ...."), the wording of that category 
must be changed. If repeated trials using other composition samples do 
not bring near consensus, the category must be discarded. 

V. Choosing the Topics 

The participants select a pair of topics on which students will write. 
Two s imi lar topics are needed because the design requires one "before" and 
one "after" com>o8ition from each student. Certain topics must be avoided, 
for exanq>le, those that invite triteness ("What are a student's [teacher's] 
obligations to his school?") and those that are biased against students who 
prefer not to reveal personal matters ("What are your inmost fears [hopes]?") 
or who may not believe the statement ("To what extent are parents [children] 
responsible for the generation gap?"). Rhetorical devices should not be 
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prescribed beceuse "compare end contrast" or "analyze end discuss’ 9 may be 
veriously interpreted. 

VI. Instructions to Students 

Instructions cen be kept to e minimum because the only really impor- 
tant limitation on the student is the time he spends in writing the compo- 
sition. Simply distributing bluebooks with the printed directive, "Write 
a composition on the topic.... You have 50 minutes," is actually suffi- 
cient for purposes of the study. Other directives may be employed ("Write 
on one side of the paper only." "Use blue ink.") but care should be taken 
that they do not confuse the student or bias the results. Data on individ- 
ual students may be gathered by means of a self-report form [Figure 1] . 

Implementing the Study 

I. Co-ordinating the Study 

The participants may select any outsider such as the college research 
director or a departmental secretary as project coordinator responsible for 
preparing the bluebooks and distributing them to the participants. He also 
collects and codes the compositions after they have been written. 

II. Administering the Compositions 

During the first week of his course, each participating instructor 
randomly distributes to his class or classes a number of bluebooks with 
the notation, "Wtite on topic 'X'" and an equivalent number that say 
"Write on topic 'Q'". After the students write on the designated topics, 
the bluebooks are collected and sent to the coordinator along with a cover 
letter noting pertinent data about each class. [Figure 2.] 

At the end of the course, each student receives a bluebook with his 
name and the directive, "Wtite on topic 'X'" (if he previously wrote on 'Q') 
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Code Number 

(LEAVE BLANK) 

INSTRUCTIONS TO STUDENT 

1. Fill In the form below: 

Name 

LAST NAME FIRST NAME MIDDLE INITIAL 

School 

Cour s e 

Date 

Sex: Male Female 

Have you a high school diploma? 

Tea No 

Have you attended any college 

prior to this tem? 

Yes No 

Your age: (Check one) 

under 17 17 18 

15 20 21“ ~22 23-26 27-30 *31-35 36 

2. Write a composition in this bluebook. 

Write in ink on one side of the page only. 

Write on alternate lines. 

You are to write on the topic: (to be selected 

by participating instructors) 







FIGURE 1 



r^~^v.»ys* ^>vw-^, v>^ - ^ *• >vi«723v?*^ .«<■*# v>i<, j^^vVv^' <r^^v»^aj*^ 



CLASS INFORMATION SHEET 

TEACHER: PLEASE COMPLETE AND ENCLOSE ONE COPY OF THIS FORM FOR EACH CLASS : 

YOUR NAME 

WHAT IS THE LEVEL OF THIS CLASS? 

(English 1 - first semester freshman reading and composition, ) 
(see below, at Superior College. This is the highest of three) 
(tracks . ' 

WHAT SCREENING PROCEDURES AND CRITERIA WERE USED TO PLACE STUDENTS 
IN THIS CLASS? 

(SCAT test - Student must have scored in the 50th percentile ) 
(or above. ) 
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or ,s Write on topic 'Q' (if he previously isrote on ’X')". Thus, each stu- 
dent writes on both topics, preparing one composition before instruction 
begins, the other at the end of the course. 

III. Coding 

The coordinator remover* all identifying marks from each composition. 

He enters its author’s name on a list and assigns a code number to it. 

Code numbers are recorded on a master sheet. Care must be taken that num- 
bers do not reveal the time when — or by whom — the composition was 

written. 

IV. Sampling 

Sampling is desirable if the total number of compositions to be scored 
exceeds 60 per participating instructor (30 written "before" and 30 "after" 
instruction). If the sampling procedure is used, a number of "before" com- 
positions are drawn at random. After all compositions have been collected, 
only those written by students in the sample group are kept; the others are 
discarded. Size of the sample will vary but it should include not fewer 
than one-third the compositions from a single class. 

V. Distributing the Compositions for Scoring 

The number of compositions scored by each participating instructor 
depends on the total number of participants. For example, if there are 60 
compositions in the sanple and 2 participants, each scores 30; if there 
are 100 compositions and 5 participants, each scores 20. 

When the condos itions are ready to be scored, they are stacked accord- 
ing to the class in which they: were written. "Before" and "after" composi- 
tions are mixed together. A number is drawn from each stack in a manner 
that each participating instructor gets an approximately equivalent number 
to score from each class. 
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VI. Scoring 

Each participant scores the compositions as distributed by the coordi- 
nator. He follows the key that he had earlier helped develop. A separate 
scoring sheet based upon the key is used for each composition. [Figure 3.] 
VII. Collating Results 

Score sheets are collected and tallied [See Figure 4] . For each 
student and for classes, the data sheet should include room for other 
pertinent information such as student's prior grade point average, age, 
etc. depending on the questions to be asked of the data. Hand. tabulation 
is sufficient for most purposes; that is, computer analyses are not nec- 
essary. 

Analysing the Data 

Individual scores on the first composition only can answer questions 
such as: 

1) Is there any relationship between a student's ability to write 
at entrance and his first term grade point average? 

2) To what extent do English placement procedures actually relate 
to writing ability? 

3) To what extent is writing ability related to scores made on 
entrance examinations or intelligence tests? 

4) Do students who score; low in any one area tend to score low 
in every area (content, organization, mechanics)? 

5) Is there any relationship between a student's writing ability 
and his persistence to the end of the course? 

Comparison of group means can answer questions such as: 

6) In what ways do students enrolled in "remedial" courses write 
differently from those enrolled in "regular" college courses? 

7) Is the ability to write related to students' indicated choices 
of academic majors? 

8) In what ways does writing compare between the same level classes 
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Content I. 



Organization II. 



Mechanics III. 



MOTE: 




This scoring key has been reproduced to show format only. Each group 
of instructors should develop its own key and its own categories. 

SCORE SHEET 



YES NO 



1. 


Ideas themselves are insightful. 


j 2. 


Ideas are creative or original 


3. 


Ideas are rational or logical. 


_ 4. 


Ideas are expressed with clarity. 


5. 


There is a thesis. 


„ 6. 


Order of thesis idea is followed throughout 
the essay. 


7. 


Thesis is adequately developed. 


8. 


Every paragraph is relevant to the 
thesis. 


9. 


Each paragraph has a controlling 
idea. 


10. 


Each paragraph is developed with 
relevant and concrete details. 


11. 


The details that are included £?e 
well ordered. 


12. 


There are many misspellings. 


13. 


There are serious punctuation 
errors . 


14. 


Punctuation errors are excessive. 


15. 


There are errors in use of verbs. 


16. 


There are errors in use of pro- 
nouns. 


17. 


There are errors in use of modi- 
fiers . 


18. 


There are distracting errors in 
word usage. 


19. 


The sentences are awkward. 

CODE NO. 

FIGURE 3 
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This scoring key has been reproduced to show format only. Each group j 
of instructors should develop its own key and its own categories. i 



J " W Or 5nK f, *, jp^«^ ** A v 









g 

I 



I 



fc 



8 



V 



« 

o 



bo 

c 



44 

« 



g 



8) 



4 J 

g 

3 

4 J 

« 



M 

o 

44 



44 

0) 

« 

CO 



g 






? 



f 



•I 



m 

§ 

4 J 



4 

5 



o 

M-l 



co 



0) 

s 

60 



4 ) 

4 ) 

CO 



00 

nP 

as 



& 



•• 

TJ 

o 



u 

V 

04 



4J 

o 

9 

M 

44 

« 

c 



4 ) 



ERIC 



L 



'ScIflUffllfflifilTJillMJ 



A 



• • 

g 

4 J 

8 

u 

4J 



I 



s ? 


•• 


5> co 


l-i 


o 


,• 


H 


> 


to •• 


« 


• 


M 


60 


» 


tt 


H 


• 


H 


« 


O 


H 


o 


O 



•• 

§ 

44 



c 

o 



9 

V 



U 

o 



43 

o 

« 

V 



g 



44 

c 



I 



1 



4 J 

c 

41 

•O 

9 

4 J 

CO 



•H *0 



5 



•g 

«< 



4 J 

o 

as 



R 

O 



•o 

« 

5 



I 



»4 

« 



g 

4 J 





• • 

« v 

4J 1 
1 44 

m • 

H O 
04 04 


" 4J • 

• • 

• 44 

44 1 

1 44 

• • 

£ £ 


it 'm'~ 

• m 
m » 

4t 1 
1 41 

88 
04 04 


44' m~' 

m m 

9 44 
44 l 
1 4t 

• • 

££ 




« ■' * 

• • 

• 44 
44 1 
1 44 

* 5 

*£ 


ON 


H 


r-t 


4-1 


H 




H 


H 


o 


O 


o 


O 




O 


00 


H 


H 


H 


H 






H 


O 


O 


H 


O 




H 


f^ 


o 


9 


H 


O 




H 


H 


o 


o. 


H 


o 




H 


vO 


H 


H 


o ■ 


H 




co 


H 


H 


O 


o 


H 




CM 


in 


• ^ 


H 


o 


o 




CM 


H 


o 


H 


o 


H 




CM 


si* 


H 


o 


H 


O 




CM 


H 


© 


. o 


o 


o 




O 


CO 


o 


H 


H 


H 




CO 


H 


o 


H 


O 


o 




H 


CM 


H 


o 


H 


o 




CM 


H 


o 


o 


O 


. o 




O 


H 


H 


H 


O 


H 




CO 




o 


o 


H 


o 




H 


o 


H 


H 


H 


H 








H 


O 


O ’ 


. o 




H 


ON 


o 


o 


H 


H 




CM 




o 


o 


© 


O 




o 


00 


o 


o 


o 


o 




O ■ 




o 


o 


o 


. o 




© 


f* 


H 


H 


H 


»“4 








o 


H 


H 


o 




CM 


\o 


H 


H 


© 


H 




CO 




H 


H 


o 


f-4 




CO 


m 


H 


H 


o 


H 




CO 




o 


O 


o 


O 




o 


< 


H 


H 


H 


o 




CO 




© 


H 


o 


o 




«H 


CO 


© 




H 


H 




•0 




o 


o 


o 


o 




© 


CM 


H 


o 


H 


H 




CO 




o 


o 


H 


H 




CM 




H 


6 


o 


H 




CM 




H * 


o 


o 


O 








* 














‘5 

- 


~ 








• *• 


j 

8 


« 

■a 


• 

1 


1 


• 

| 




iH • 

13 


4J 

! 

< 

i 


m 

\ 




! * 


1 * 




Is 



g 

4i 



g 



s 



|4 

V 



o 

3 



0 

1 

2 



b 

1 



« 



i 



41 

s 

1 



i 






j 






i 



3 



i 







11 



In the iiie school ? In different schools (if Instructors fros 
more then one Institution ere involved)? 

Comparison of change scores can be used to answer* 

9) How much does student writing Improve from the beginning to the 
end of a course as indicated by the two compositions? 

10) What kinds of improvement are made? 

11) Do "remedial" courses remedy certain types of writing deficiencies? 

12) Can change in student writing be traced to different instructional 
procedures employed in one or another course section 

y 

In the sample tally sheet reproduced in Figure 4, Hiss Doe*s Honors English 
Class of four students wrote the essays as directed. She wanted an answer to 
Question 10: What kinds of improvement are made? 

Results showed that her students did not improve their ability to make para- 
graphs relevant (Category No. 8 on the Score Sheet) or to avoid errors in the use 
of modifiers (Category No. 17). However, they showed gain in many categories, 
particularly their ability to make ideas rational (No. 3), state a thesis (No. 5), 
develop each paragraph (No. 10), and avoid errors in word usage (No. 18). Thus, 
she is aware of the types of improvement made by her students and can modify her 4 

* 7 

instruction accordingly. She and her colleagues may answer several other ques- 
tions by checking the tally sheets against grade marks, entrance exams, and so on. 

I 

Discussion ! 

^ t 

Implications of using the writing samples to answer those types of questions 

l 

•a 

seem obvious. . For one thing, changes in curricular emphases or instructional pro- j 

cedures may be undertaken and the study repeated to determine if "innovation" is 
in fact effectual. When reliable data on student gain in writing ability are 

1 

produced, questions regarding the relative value of traditional grammar, struc- 
tural linguistics, and other course emphases may be given a common referent. 
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And the worth of instructional procedures such as writing laboratories may be 
assessed according to the degree of improvement that appears in student writing* 

A faculty that would prepare instructional sequences for reducing student writ- 
ing deficiencies could likely think of many other uses to which the design can 
be used to advantage. 

The two most important characteristics of the design itself are 1) it pro- 
vides for multiple blind scoring and 2) categories on which compositions are 
scored are developed by the participants. The reason for having had the students 
write. On topic *X* at the beginning and on *Q* at the end (and vice versa) Is 
that when participating instructors score the compositions, they must not know 
whether the sample they are reading was written early or late. In that way, un- 
intentional biases such as, "You can't expect a beginning student to know that" 
and "He should have learned that in the course" are mitigated. The design pro- 
vides for blind scoring not only in terms of students' names and classes but also 
in point of time when the paper was written. Without a blind scoring feature, 
any design that depends on "subjective judgment (as it must if a composition scor- 
ing device is to be accepted by a faculty)" suffers. Another value of this de- 
sign is that assigning papers for many classes to be read by many scorers allows 
each paper to be read but once while distributing the influence of any single 
reader. 

The fact that the scoring key is developed by the participants is no less 
signif leant, the worth of any scheme is lessened to the extent instructors fail 
to understand or to agree with the categories suggested. When participants build 
their own device, try it out and revise it so that misunderstandings are lessened, 
the tool becomes much more useful. The degree of confidence one wishes to place 
in the findings is directly related to the extent to which he accepts the whole 
group's ability to judge compositions accurately. 
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A single English department may wish to use the design for purposes of in- 
tragroup comparison of Instructional effect or for use in suggesting student 
placement in courses* In that case all members of the department would partici- 
pate in developing the scoring key which could be built in a couple of two hour 
meetings. If the group of participants were larger than a single department, 
some form of conference or workshop covering a day or two would have to be ar- 
ranged. It is likely that a period of time is required to bring simplest under- 
standing of words used when people previously unknown to each other are brought 
together. The farther members of the group are away from each other in terms 
of departmental or institutional affiliation, the longer the time required to 
develop the scoring key. Members of a single department often know — although 
they may not agree with — what their fellows mean when they use particular 
terms. 

The scoring key should be based on a large number of categories so that each 
point of inclusion can be assessed on an "either/or" or "yes/no" basis. If few 
categories are used, the tendency will be to scale quality of response on an 
"always / sometimes /never " basis or on a five point scale. However, such scaling 
makes scoring more complex, and, most important, lends little direction to the 
way instructional procedures might be modified to overcome discerned deficiencies. 

Composition scoring designs have never been popular. This is due in large 
measure to the field's continued acceptance of standards that shift in accordance 
with any instructor's whims — a pernicious interpretation of "academic freedom" 
— and to a persistent belief that ones own assessment of his students' writing 
is the only worthy measure. Many instructors do not agree with their colleagues 
even on criteria to be employed in viewing students' work, let alone on their 
evaluations of it* Too, the idea of "scoring" a composition is reprehensible to 

l 

many even though they in fact do not hesitate to mark errors, make comments about 



the worth of passages, and assign overall grade narks to student papers. 

An English faculty that would Inprove Its Instructional procedures so that 
learning Is effected can well enploy a design such as the one promulgated here. 
It falls to shed light on the Issue of how ouch a student's Inner being has been 
affected by his college experience but It does provide one answer to the ques- 
tion. "Is anyone learning to write?" 
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